3.4. Memory

What does "memory" mean in this course?

The word covers different stores that should not be conflated:

This page covers long-term memory and knowledge retrieval. A runbook is not automatically trusted simply because it is stored locally; its content is still data the model must cite and policy must constrain.

What should an agent remember across sessions?

Facts the next conversation needs and cannot recompute: the incident under investigation, remediation already attempted, and outcome notes. In-session history dies with the session, and runbook retrieval only knows the knowledge base. longterm.py gives the Ops Copilot two tools — save_incident_note and recall_incident_context — backed by a small SQLite table in the disposable state directory. Notes are isolated per user and pass PII redaction before persisting:

# Memory is a persistence boundary: redact before the write, not after the read.
redacted = redact_pii(cleaned)

Memory is a data store like any other: it never touches the seed dataset, and mise run data:reset clears it along with the rest of the runtime state.

The cross-session boundary depends on a stable ToolContext.user_id. The default unauthenticated A2A adapter derives A2A_USER_<context-id>, so a new A2A context (including a browser reload in the minimal client) is a new logical user and cannot recall the previous context's notes. A shared or production client must authenticate the caller and propagate a durable subject before treating this store as human-level long-term memory.

Why make memory explicit tools instead of automatic?

Silent context stuffing hides reads and writes from everyone debugging the agent. As tools, every recall and save appears in the trace, can be audited, and can be tested deterministically offline. Explicit tools also keep the boundaries inspectable: input validation (normalize_incident_id, a length cap, no empty notes) runs at the tool edge, and the redaction step above is a visible policy, not a hope. The cost is that the model must decide to call recall_incident_context — the tool docstring tells it to do so at the start of an investigation, and trajectory evaluation is where you verify it does.

How is a known runbook fetched?

An incident carries a validated runbook slug. The tool normalizes it before any path lookup:

def get_runbook(slug: str) -> dict[str, Any]:
    """Fetch a runbook by its exact slug (e.g. an incident's ``runbook`` field).

    Args:
        slug: The runbook identifier, e.g. ``high-latency`` or ``service-down``.

    Returns:
        A dict with the ``slug`` and its markdown ``content``, or an ``error`` if unknown.
    """
    # Parse at the boundary: normalize once here, then work only with the trusted
    # value — read, error message, and result all speak the normalized slug.
    normalized = normalize_slug(slug)
    if normalized is None:
        known = ", ".join(data.list_runbook_slugs())
        return {"error": f"Invalid runbook slug {slug!r}. Available runbooks: {known}."}
    content = data.read_runbook(normalized)
    if content is None:
        known = ", ".join(data.list_runbook_slugs())
        return {"error": f"No runbook named {normalized!r}. Available runbooks: {known}."}
    return {"slug": normalized, "content": content}

The exact excerpt is build-checked against memory.py. normalize_slug permits only lowercase alphanumeric kebab-case, and only the trusted normalized value reaches the data layer. ../../secret never becomes a filesystem path.

How does free-text retrieval work?

search_runbooks uses deterministic term scoring:

• Drop tokens shorter than three characters.
• Count how many documents contain each term.
• Weight rare terms more highly than ubiquitous terms.
• Add a strong boost when a term matches the runbook slug.
• Sort by score descending, then slug ascending for stable ties.

scored.sort(key=lambda row: (-row[0], row[1]))

The result is inspectable, fast, cheap, and reproducible. It is intentionally not described as semantic search — that is the opt-in branch below.

When do embeddings beat keyword retrieval?

When queries paraphrase the documents instead of quoting them — an engineer types "checkout is slow" while the runbook says "high latency". Whether that matters for this corpus is a measurement, not a fashion choice: enable semantic retrieval only if the evaluation below shows it beating the keyword baseline on this dataset.

The implementation in retrieval.py stays minimal and account-free: sqlite-vec stores runbook-chunk vectors in the state directory, and nomic-embed-text runs through the local Ollama embeddings endpoint. Chunking is heading-bounded so each vector covers one procedure step or symptom block:

def chunk_runbook(slug: str, content: str) -> list[str]:
    """Split one runbook into heading-bounded chunks, each prefixed with its slug."""
    pieces = [piece.strip() for piece in _HEADING.split(content) if piece.strip()]
    return [f"{slug}: {piece}" for piece in pieces] or [f"{slug}: {content.strip()}"]

The option is off by default (AGENT_SEMANTIC_RETRIEVAL=false), which keeps the offline test gate deterministic and model-free. When enabled, search_runbooks ranks by embedding similarity instead; when the embedding endpoint is down, it falls back to the keyword scorer with an explicit warning log — never a silent degradation. Own the failure modes before trusting the upgrade: an empty result set, a confidently-wrong nearest neighbor (cosine distance always returns something), and the endpoint outage above.

How do I evaluate retrieval quality?

With ground truth the dataset already contains: every incident names its runbook, so hit-rate@k needs no hand labeling. evals/retrieval_eval.py builds a query from each incident's title and summary, scores both retrievers at k=1 and k=3, logs the comparison to MLflow, and prints a verdict:

cd agents/python
mise run eval:retrieval

Measured checkpoint (2026-07-12, dataset commit ad4854b, Ollama 0.31.2, nomic-embed-text blob sha256:970aa74c0a90):

On these ten incident queries, semantic retrieval loses at k=1 and only matches at k=3. The evidence therefore supports the shipped default: keep semantic retrieval disabled for this small, vocabulary-aligned corpus. Re-run the checkpoint when the incidents, runbooks, chunking, or embedding model changes; this snapshot is a versioned observation, not a universal ranking claim.

def hit_rate(retrieve, k: int) -> float:
    """Fraction of incidents whose runbook appears in the retriever's top-k."""
    pairs = cases()
    hits = sum(expected in retrieve(query, k) for query, expected in pairs)
    return hits / len(pairs)

The semantic side calls the local embeddings endpoint, so this evaluation stays outside the offline test gate. Set AGENT_SEMANTIC_RETRIEVAL=true only if the semantic hit rate beats the keyword baseline here — and remember what you now own: embedding model/version, chunking, index rebuilds, deletion, poisoning defenses, and re-running this evaluation when the corpus changes.

The first CPU request may need to load the embedding weights from disk. AGENT_EMBEDDING_TIMEOUT_S bounds that cold start separately (120 seconds by default), so ordinary MCP tools retain their tighter deadline.

How do you defend against retrieval injection?

• Index only reviewed sources and preserve provenance.
• Treat retrieved instructions as lower priority than system/runtime policy.
• Never let a runbook expand the tool allowlist or self-approve an action.
• Return/cite the source slug so a human can inspect it.
• Bound result count and content size.
• Include malicious or contradictory documents in adversarial tests.

What is the retrieval checkpoint?

cd agents/python
uv run pytest tests/test_memory.py tests/test_longterm.py tests/test_retrieval.py

Verify exact-slug lookup, deterministic tie ordering, no-match behavior, non-positive limit handling, path traversal rejection, per-user note isolation, redaction before persistence, and the keyword fallback when embeddings are unavailable. Then manually compare the top result for database connection pool exhausted with the runbook source.

What is actually inside my agent's context window?

Everything the model sees each turn is assembled by the runtime, and all of it competes for the same window — on the local path, the modest usable context of Qwen3-4B through Ollama. For the Ops Copilot, one request is composed of:

The tool schemas are easy to underestimate: the agent registers a dozen tools (four reads, two knowledge, two guarded actions, two memory, two skill tools), and each docstring you write for the model is prompt text on every single turn. Tool results are the other heavyweight — search_runbooks returns up to three whole runbooks as markdown, and once returned they live on in the session history.

How do I measure tokens per component?

Per turn, you do not have to guess: the after-model callback in budget.py reads usage_metadata from each response, accumulates it in session state, and emits the running totals as agentops.tokens.session.* span attributes plus an agentops.tokens counter that Prometheus scrapes — the same telemetry that backs cost attribution in 7.3. Costs. Watching prompt_token_count grow turn over turn shows exactly how fast history accumulates.

Per component, there is no shipped breakdown — measure by ablation. Hold one question constant (What is the status of the checkout service?), change one component, and compare the input tokens of the first turn:

1. Switch the six read/knowledge tools between local and the MCP route (AGENT_MCP_URL) to see the schema cost of each toolset.
2. Shorten one tool docstring and diff the input count.
3. Ask a question that triggers search_runbooks versus one answered by a single status read to see retrieval weight.

Reset state between measurements (mise run data:reset), and remember the gateway streaming path reports no usage — measure on the default non-streaming path.

What do I drop first when I run out?

In order of least behavioral risk first, all demonstrable on the local stack:

1. Retrieval width. search_runbooks(query, limit=3) returns whole documents; the model rarely needs three runbooks to cite one. The semantic option retrieves heading-bounded chunks instead of full files, which is itself a context reduction.
2. Tool schema weight. Terser docstrings and fewer registered tools shrink every turn. Note that routing reads through MCP does not make their schemas free — discovered declarations still enter the context; the saving comes only from what you stop exposing.
3. Session history. The course ships no automatic summarizer, so the honest lever is deliberate: end the session, after saving durable findings with save_incident_note, and let recall_incident_context carry them into the fresh session — long-term memory used as explicit history compression. Skills already apply the same idea: list_skills shows only names and descriptions until load_skill pulls one body in.

Every cut trades against quality: a terser docstring can cost a correct tool choice, a smaller limit can drop the right runbook. The eval set in 4.4. Evaluations is the check — re-run the trajectory scores after each reduction and treat a drop as the real price of the tokens you saved.

Why does my agent silently forget the start of a long conversation?

Because the model's advertised maximum and the serving window are two different numbers. ollama show qwen3:4b prints the architecture's context length, but the Ollama server applies its own configured window — a few thousand tokens by default unless OLLAMA_CONTEXT_LENGTH raises it. When the composed prompt (instruction, tool schemas, full history, tool results) exceeds that window, Ollama truncates the oldest tokens and answers anyway: the agent process receives no error, only a model that has lost the earliest content. Since the instruction sits at the front, a long session can shed the operating rules themselves — grounding and approval discipline degrade exactly when the conversation is longest.

Detect it locally: the per-session input-token attributes from the telemetry above keep climbing while the Ollama server log reports the truncation, and behavior drifts (uncited claims, forgotten constraints from early turns). Fix it explicitly: raise OLLAMA_CONTEXT_LENGTH to what your hardware's memory allows, shrink the context with the levers above, and set AGENT_MAX_TOKENS_PER_SESSION so a session ends with an actionable message before it degrades silently.